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STATE -LEVEL MEASUREMENT OF PERFORMANCE OUTCOMES 
IN POSTSECONDARY VOCATIONAL EDUCATION 



Volume I: Executive Summary 
An Overview of State Policies 



EXECUTIVE SUMMARY 

In vocational education, as 1n other areas of education, 
assessment of performance outcomes Is Increasingly viewed as a way to 
measure program quality and to motivate program Improvement. This 
study surveyed states to learn more about how they currently assess 
student outcomes 1n postsecondary vocational programs. It was found 
that most states have undertaken limited activities to assess 
performance outcomes, but few states have employed multiple measures of 
performance, and most have had difficulty 1n analyzing the data that 
they collect or 1n using 1t for program Improvement or policy. It was 
also found that many states are making efforts to increase the quality 
and comprehensiveness of the information collected and to make greater 
use of the Information collected. 

Fedaral education policy H moving rapidly toward an emphasis on 
measuring performance as an Indicator of program accountability. The 
recently reauthorized Carl Perkins Vocational Education Act requires 
states to develcp performance measures and standards for vocational 
education at the secondary and postsecondary level s. The provisions of 
the Student Right- to-Know and Campus Security Act of 1990 mandate that 
Institutions offering postsecondary vocational programs provide 
prospective students with Information about the graduation rates of 
previous students. This 1s commonly called "consumer rights 
Information." Performance standards, tied to accreditation or 
eligibility for student aid, is Increasingly viewed as one option to 
curt program abuse and Improve accountability in postsecondary 
vocational programs. 

This study was motivated by the "consumer rights" rules in the 
U.S. Department of Education's student aid crogram; hence, the focus 1s 
on postsecondary education. The study offers a "snapshot" of what 
states were doing in the spring of 1990 to collect Information on 
program outcomes for students enrolled in postsecondary vocational 
education. It attempts to answer three questions: 

1. To what extent do states collect program Information on student 
outcomes and, if so. what types of information? Do outcome 
measures include program completion, job placement, training- 
related job placement, earnings, job competencies, or 
satisfaction of occupational licensing requirements? 
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2. How is Information on program outc omes obtained? What 1s the 
source of data? Who Interprets the data? What are potential 
threats to the accuracy, comparability, and consistency of the 
Information obtained? 

3. How is Information on program outcomes used? Are there state 
standards that must be satisfied? Is the information used to 
allocate funds for program management or policy? Is the 
information made available to consumers? 

Information for this study was collected during the spring of 
1990. A multi-stage process, beginning with a review of state reports 
and legislation, was followed by telephone interviews with state 
officials to get additional information. To complete a comprehensive 
description of the activities in each state, it was necessary to 
contact a wide range of state officials responsible for postsecondary 
vocational education in the public and private nctors. The average 
number of respondents per state was five. (It was also necessary to 
talk briefly with an average of four additional persons in the state 
government to Identify the mcst knowledgeable Individuals.) Efforts 
were made to validate respondent information about the nature of state 
policy by seeking confirmation from more than one source or from 
written documentation. This was not always possible. The practices 
reported from the states may therefore reflect some degree of 
overstatement. 



Major Findings 

Governance 

There is typically no single state agency with responsibility for 
vocational education. Governance of secondary and postsecondary 
vocational education is commonly divided among a variety of state 
agencies. Within the postsecondary sector, there may be different 
agencies responsible for programs offered through vocational -technical 
institutes and community colleges. Almost always, proprietary schools 
are overseen through separate state entities, and frequently by 
multiple licensing boards corresponding to different occupational 
areas. 

The division of vocational education governance severely limits 
the ability of states to develop comprehensive policies that apply 
equally to all institutions, public and private, offering postsecondary 
vocational training. Only in a few states, all of which have a single 
agency administering all of postsecondary vocational education, have 
there been efforts to implement uniform practices across all sectors of 
postsecondary vocational education. 

This study found no state that currently measures program 
outcomes in the same way for students in public postsecondary programs 
and in proprietary school programs. In interpreting the results of 
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this study, therefore, 1t 1s Important to remember that findings about 
state practices are characterizations about particular sectors of 
postsecondary vocational educat1on--not all sectors that provide 
vocational education. 

Moreover, to the extent that state policies exist, the most 
ambitious efforts have been undertaken 1n public vocational education. 
Forty-six states collect some outcome information from public 
Institutions, while only 29 collect any outcome Information from 
proprietary institutions. In all but 7 states, more types of 
Information are collected from public Institutions than from private 
institutions. 



Tvoes of Information Collected 

Most states collect some information about program outcomes. The 
most frequently collected outcome measure is the number of students who 
complete a training program (found in 46 states for public schools and 
29 states for proprietary schools). The next most frequently. collected 
measure 1s whether program completers are placed 1n jobs (33 states for 
public schools and 21 states for proprietary schools). Most states 
attempt to determine whether the job placement 1s related to the 
training provided, although this practice 1s much less common for 
proprietary schools than it 1s for public schools. Outcomes such as 
earnings, gains in basic skills and job knowledge, and results from 
state licensing exams are collected in a small number of states. 



Definitions 

Most states that measure completions have developed standardized 
definitions of who is a completer (39 of 43 states in the public sector 
and 26 of 29 states for proprietary schools). The most common 
definition of a completer is an individual who completes all program 
requirements and is awarded a degree, diploma, or certificate. 

In 13 states, one critical limitation to the definition of a 
program completer Implemented for public Institutions 1s that 
Information is collected only for recipients of associate degrees. As 
almost all other measures collected are applied only to program 
completers, students attending a large segment of public postseco;idary 
vocational education are excluded from the information system 1n these 
states. No parallel limitations were found among definitions for 
proprietary institutions. 

Somewhat fewer states have standardized definitions for measuring 
job placement. All states that count placements limit the data 
collected to program completers. The most common means for determining 
whether placement is "training related" is for the students themselves 
to determine whether their employment is related to the training 
received. Ten states ask the public sector institutions to make this 
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determination, and 7 states permit proprietary schools to decide which 
job placements are training related. Only 1n Arkansas (for voc-tech 
schools), Florida, Minnesota, and Washington 1s the determination of 
"training relatedness" made at the state level. 

Computation of Rates 

The computation of a completion or placement rate requires at 
least two elements: 1) a definition of a completer or a job placement, 
and 2) agreement on the base to which the number of completers or 
placements 1s to be compared. Establishing a base number for these 
rates and obtaining appropriate data appear to be areas where states 
are experiencing difficulty. Of those states that collect completion 
data, 75 percent report that they compute a completion rate for public 
vocational education and 67 percent report that a completion rate is 
computed for the proprietary component. One area of difficulty 1s 
determining which categories of students to consider program enrol lees 
(e.g., full-time, part-time, those completing a minimum portion of 
total course work). 

The computation of placement rates Involves greater difficulty, 
with only about half the states that collect placement data actually 
computing a placement rate. The primary difficulty is 1n determining 
which categories of unemployed persons to exclude from the calculation 
(e.g., those not seeking employment, or who do not use the school's 
placement service). 



Data Source s 

The most frequently used source of information 1s surveys of 
former students; the second most popular method 1s surveys of 
employers. A few states use third-party data, such as state 
unemployment Insurance wage records. There was reported to be 
increasing interest in the use of these third-party sources, with the 
goal of reducing the burden on schools and Increasing the reliability 
of the Information obtained. Twenty-one states require public 
postsecondary vocational schools to conduct graduate surveys and to 
tabulate and report the results. In Arkansas and Washington, these 
surveys are conducted directly by the state oversight agency for public 
vocational programs. Only Indiana currently requires proprietary 
schools to survey their graduates. Several additional states will soon 
mandate surveys for graduates of public and proprietary vocational 
programs . 



Data Verification 

Few states in which performance data are collected attempt to 
verify the validity of the data reported by- schools. A few states 
contact a small sample of students or employers directly to validate 
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claims made by schools. More typically, states simply review 
Information maintained in school files to see whether it matches data 
reported to the state; the states make no independent effort to 
determine the validity of this information. 



Analyses 

Analysis of the data is limited. Most states that require 
schools to report data do not actually analyze it. The states that do 
conduct some analysis tend to focus on comparisons of key indicators 
with established state standards, such as a minimum placement rate, or 
on the construction of a profile by occupational program. These 
profiles may then be scrutinized to determine whether any programs are 
performing unusually well or poorly. 



Uses 

States that collect and analyze student outcome data use this 
information in a variety of ways. Thi most frequent use of the outcome 
measures collected is to conduct individual program reviews. Another 
major use is to respond to ad hoc requests from the governor's office, 
the legislature, or from other government agencies. Fourteen states 
compile profiles of outcomes for different occupational programs and 
make this available to public institutions; 8 states do the same for 
proprietary schools. 

Fourteen states report that the information collected from 
programs in public institutions is made available to consumers. Four 
of these states also make information from proprietary schools 
available to assist consumer decision making. Four additional states 
make proprietary school information available exclusively. Tennessee 
uses performance -based data to allocate supplemental funds to public 
postsecondary Institutions. 

Efforts to make information available about performance to the 
public range from passive responses to requests to mailings to high 
schools and public libraries. No states reported publishing results in 
the newspaper. 



T rends 

State policies on outcome measurement are changing rapidly. All 
states that collect some program performance data have altered some 
aspect of their policies regarding assessment of student outcomes in 
postsecondary vocational programs during the past 2 years. Overall, 19 
states either initiated the collection of outcome information or made 
m?jor changes to increase the amount of information collected from 
schools. Four states made changes for both public and proprietary 
institutions; 11 states addressed only public institutions; and 4 
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states addressed proprietary schools only. Seven states have passed 
legislation that has not yet been Implemented. Other states have 0111s 
pending or pilot projects underway. In 15 states there was no activity 
identified that would result in a change for public institutions, and 
In 31 states there was no report of change for private institutions. 



This document presents an overview of the findings of this study 
effort, including synopses of practices and major trends, and a summary 
and conclusions section. A companion document, Volume II: Appendix: 
teflles of State Data Collection ArHvitie^ presents more detailed 
findings on a state-by-state basis of an overview of individual state 
activities in the measurement of the outcomes of postsecondary 
education. 
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STATE-LEVEL MEASUREMENT OF PERFORMANCE OUTCOMES 
IN POSTSECONDARY VOCATIONAL EDUCATION 

Volume I: Executive Summary 
An Overview of State Policies 

I. INTRODUCTION 

In vocational education, as 1n other areas of education, 
assessment of performance outcomes 1s Increasingly viewed as a way to 
measure program quality and to motivate program Improvement. This 
study presents an overview of how states currently assess performance 
outcomes 1n postsecondary vocational programs. 

Federal education policy is moving rapidly toward an emphasis on 
measuring performance as an Indicator of program accountability. The 
recently reauthorized Carl Perkins Vocational Education Act requires 
states to develop performance measures and standards for vocational 
education at the secondary and postsecondary levels. Current U.S. 
Department of Education regulations for the Stafford Student Loan 
Program mandate that Institutions offering postsecondary vocational 
programs provide prospective studants with information about the 
graduation and job placement rates of previous students. This Is 
commonly called "consumer rights information." Performance standards, 
tied to accreditation or eligibility for student aid, is Increasingly 
viewed as one option to curb program abuse and improve accountability 
1n postsecondary vocational programs. 

This study was motivated by the "consumer rights" rules 1n the 
Stafford Loan Program, recently supplanted by the Student Right-to-Know 
and Campus Security Act of 1990, which has similar provisions regarding 
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the collection and disclosure of completion rates of former students. 
Hence, the focus Is on postsecondary education. The study of the 
governance of Individual boards for licensed occupations was beyond the 
resources available for this study, so little specific Information 1s 
available here about institutions specializing in such fields as 
nursing, allied health, cosmetology, or real estate. 1 

The study offers a "snapshot" of what states are currently doing 
to collect information on program outcomes for students enrolled 1n 
postsecondary vocational education. It attempts to answer three main 
questions: 

1. To what extent do states collect program Information on student 
outcomes and, If so, what types of Information? Do outcome 
measures Include program completion, job placement, training- 
related job placement, earnings, job competencies, or 
satisfaction of occupational licensing requirements? 

2. How is information on program outcomes obtained? What is the 
source of data? Who collects it? Who interprets the data? What 
are potential threats to the accuracy, comparability, or 
consistency of the information obtained? 

3. How is infon tlon on program outcomes used? Are there state 
standards tha must be satisfied? Is the Information used to 
allocate funds for program management, or policy? Is the 
information made available to consumers? 



Efforts were made during the data collection to include information on the 
licensed trades, such as cosmetology, real estate, and truck driving. During 
c r>versations with the licensing boards of the first states contacted, no board 
was found that collected any information other than success on the licensing 
examination. Although some used this information in determining whether to 
license a school or program for operation, none of the contacted boards 
reported sharing their information with the public for consumer rights 
purposes. Because there are so many licensing boards in the typical state, a 
decision was made that it would require too much of the project resources to 
contact all licensing boards in every state. Instead, project staff called on two 
licensing boards in a random sample of states. None revealed practices that 
differed from the licensing boards in the initial states. 

2 



The information for this study was collected during the spring of 
1990. A multiple- stage process, beginning with a review of state 
reports and legislation, was followed by telephone interviews with 
state officials to fill in missing information. To complete a 
comprehensive description of the activities in tach state, it was 
necessary to contact state officials responsible for different aspects 
of postsecondary vocational education in the public and private 
sectors. The average number of respondents per state was five. 
Efforts were made to validate respondent information about the nature 
of state policy by seeking confirmation from more than one source or 
from written documentation, but either or both of these was not always 
possible. The practices reported from the states may therefore reflect 
some degree of overstatement. 

Of the 50 states, the District of Columbia, and Puerto R1co, no 
state 2 was found to have had an outcome measurement policy in place 
that had not been changed for more than 2 years. The majority of 
states have either initiated new policies for collecting outcome data, 
have expanded or improved their collection practices, have passed 
legislation calling for changes to be implemented in the near future, 
or have introduced bills in the state legislature seeking authorization 
for similar changes. 

The overview presented in this report reveals that the states 
vary widely in the outcome measures they have selected to collect, the 
definitions of those measures, the method of collection, the analysis 

2 For convenience, throughout this report all of these jurisdictions are referred 
to as "states." 
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of these data, and the uses for which they are Intended. Although 
there has been a pace of change and a consistent movement to Increase 
the levels of activity 1n the collection of outcome measures, there are 
no identifiable patterns 1n these changes. 

This document presents an overview of the findings of this study 
effort, Including synopses or practices and major trends, and a summary 
and conclusions section. A companion document, Volume II; Appendix: 
Profiles of State Data Collection Activities , presents more detailed 
findings on a state by-state basis of an overview of individual state 
activities in the measurement of the outcomes of postsecondary 
education. 
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II. COMPARTMEKIfiLIZATIOM OF THE OVERSIGHT OF 
POSTSECOHDARY VOCATIONAL EDUCATION 

One of the most striking features of the measurement of student 

outcomes In postsecondary vocational education has been the variety of 

administrative structures used by states and that few states have 

chosen to centralize the responsibility Into a single agency. Table 

1 1 - 1 presents a brief overview of the distribution of agencies at the 

state level responsible for the oversight of the components of 

vocational education. Specific agencies for each component are shown 

In Volume II of this report. 



Table II -1. Organization or Agency Responsible for Each Component 
of Vocational Education 



ORGANIZATION 
RESPONSIBLE FOR 
ADMINISTRATION 


NUMBER (PERCENT) OF STATES ASSIGNING 
RESPONSIBILITY TO EACH TYPE OF 
ORGANIZATION 


COMMUNITY 
COLLEGES 


PUBLIC 
VOCATIONAL/ 
TECHNICAL 
SCHOOLS 


PROPRIETARY 
SCHOOLS 


Office or Division of 
Department of Education 


12 (23%) 


13 (25%) 


20(38%) 


Independent Board, 
Commission, or Council 


27 (52%) 


19 (37%) 


22 (42%) 


Higher Education or 
Community College System 


5 (10%) 


2(4%) 


1(2%) 


Office, Division, or 
Department of Postsecondary 
Education 


2 ( 4%) 


2 ( 4%) 


0(0%) 


Department of 

Vocational/Technical 

Education 


2(4%) 


5 (10%) 


1(2%) 


Licensing Board Only 


0(0%) 


0(0%) 


2 ( 4%) 


Other 


0 ( 0%) 


0(0%) 


2(4%) 


None 


4 ( 8%) 


10 (19%) 


4(8%) 
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Oversight responsibility for public and proprietary vocational 
education was found most often to reside 1n an agency Independent of 
the state department of education, such as a board of regents, 
commission on postsecondary education, or higher education council. An 
office, division, or bureau of the state department of education was 
the second most common location. 

Thinking of postsecondary vocational education as having three 
major components (community colleges, vocational -technical institutes, 
and proprietary institutions), one would find that only 8 states 
administer all three components through the same agency. In 18 states, 
community colleges and vocational -technical schools are administered by 
the same agency, although the proprietary schools are administered by 
a third (one of these states has no proprietary schools). Four states 
assigned oversight responsibility for community colleges and 
proprietary schools to the same agency, and 3 of these states have no 
vocational technical schools. Responsibility for both vocational - 
technical schools and proprietary schools are assigned to the same 
agency In 2 states. The remaining 20 states have no overlap in the 
oversight of the components of postsecondary vocational education. 

Table II-2 shows the extent of overlap in administratis 
responsibilities. It demonstrates that about 38 percent of the states 
administer each component of postsecondary vocational education through 
a separate agency, and only 15 percent of the states have chosen to 
have all components administered within the same agency. 
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Table I I -2. Patterns of Governance for Postsecondary 
Vocational Education 



GOVERNANCE PATTERN 


NUMBER (1 ERCENT) 
OF STATFS 


All Three Components of Vocational Education 
Are Administered by the Same Agency 


8 (15%) 


Community Colleges and Vocational-Technical 
Schools Are Administered by the Same Agency 


18 (35%) 


Community Colleges and Proprietary Schools 
Are Administered by the Same Agency 


4 ( 8%) 


Vocational-Technical Schools and Proprietary 
Schools Are Administered by the Same Agency 


2 ( 4%) 


There Is No Overlap in Administration 


20 (38%) 



Table 1 1 -3 shows in more detail the types of state agencies where 
the overlap in responsibilities is located. The table arrays the types 
of state agencies by the number of components of vocational education 
overseen by that agency in each state. The frequencies in the table 
refer to the number of states fitting that pattern. For example, it 1s 
shown in the first column that for 5 states, the state department of 
education has oversight responsibility for all three components; in 8 
states the department has responsibility for two components; in 14 
states for one component; and in 25 states the state department of 
education has no responsibility for assessing the outcomes of 
postsecondary vocational education programs. 




Table I I -3. Commonality of Oversight Agency 



COMPO- 
NfiNTS OP 
VOC* 

HOW 
OVERSEEN 


;OFHt» 
DIVISION 

OP 
l/GrAKft* 

MENTOF 
TION 


INDEPEN- 
DENT 

BOARD, 

vUMMJV .. 
SION, 
OR 

COUNCIL 


HIGHER 
EDUCA- 
TION OR 
VVMMU* 
NITY 

COLLEGE 

SYSTEM 


OFFICE/ 
DIVISION/ 
DEPART* 
Mcrri Or 

POSTSECON. 
DARY 

EDUCATION 


DEPART 
MENTOF 
VOCATIONAL 

EDUCATION 


LICENSE 

INO 
BOARD 

v%Mt V 

UWIUY 


OTHER 


NONE 


0 


25 


20 


46 


49 


47 


SO 


49 


37 


1 


14 


6 


4 


2 


3 


2 


3 


12 


2 


8 


16 


2 


1 


1 


0 


0 


3 


3 


S 


10 


0 


0 


1 


0 


0 


0 



As of the spring of 1990, no state collected the same Information 
from both public and proprietary schools. Therefore, there 1s 
currently no relationship between the number of agencies Involved 1n 
the oversight of vocational education in a state and the similarity of 
outcome measure practices for the different components of vocational 
education. However, personnel In a few states described either 
previous or potential efforts to Implement Identical procedures in the 
two sectors. That these states are among those that administer all 
components of vocational education by the same agency suggests that the 
more typical decentralized oversight of governance of public and 
proprietary organizations Impedes the development of comprehensive 
policies. The broad Implementation of a similar uniform approach to 
the measurement of performance will require special efforts to 
coordinate across different state agencies and organizations. 
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III. COLLECTION OF PERFORMANCE DATA 

As with the decision about how to administer the assessment of 
performance outcomes, states were found to differ considerably 1n their 
decisions about what to measure and how to conduct the measurements. 

When Interested parties contemplate the Issue of measuring the 
outcomes of postsecondary vocational education, they must make a series 
of decisions. Among these are: 

• Which outcomes should be measured? 

• What operational definition will be used for each measure? 

• W111 a standardized definition be used across all Institutions, 
or will Individual schools be free to construct their own? 

• How will the necessary Information be collected? 

• How will the collected Information be used? 

• What analyses will be conducted? 

• What reports will be generated? And, 

• Whether to disseminate the collected Information and, 1f so, how 
to do so? 

In assembling the information for this study, it became clear 
that states have made different rates of progress through all of these 
decisions and have made different decisions at each point. A 
discussion of each of the major measures currently collected by the 
states illustrates the variation in policy and practice that currently 
exists among the states. Colorado appears to have developed the most 
comprehensive set of policies and practices and has eliminated many of 
the threats to the validity of the data collected that are present in 
the practices of other states. 



Program Completion Counts 
The most basic Information collected by any state or institution 
is the number of students completing the postsecondary vocational 
education program in which they are enrolled. This would seem to be a 
fundamental measure of program performance. It is also the easiest 
information to collect as it is usually available from the 
administrative records of the educational institutions, and usually 
requires no follow-up information on the students after they leave the 
program. 

However, as shown in Table 1 1 1 - 1 , the collection of completion 
data (simply counts of completers and not the computation of a 
completion rate) at the state level is not universal, especially for 
proprietary institutions. This is indicative of the status of the 
collection of outcomes measures by the states. 



Table III-l. Institutions Collecting Program Completion Data 



NUMBER (PERCENT) OF 
STATES: 


COMMUNITY 
COLLEGES 3 


PROPRIETARY 
INSTITUTIONS 


Collecting This Information 


46 (88%) 


29 (56%) 


Not Collecting This 
Information 


6 (12%) 


23 (44%) 



Vocational-technical institutions are not included in the tables that describe 
data collection practices. The study typically found that this sector of public 
postsecondary vocational education either directly modeled the practices for 
community colleges or participated in state data collection activities for 
secondary institutions, where requirements to provide placement and job- 
related information were nonexistent. 
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Of the states collecting completion data, the majority have 
developed a standardized definition for classifying former students as 
completers or noncompletors (Table I I I -2) . Thirty-nine states have 
developed a standardized definition for public Institutions, and 26 
states have developed such a definition for proprietary schools. Six 
states are 1n the process of developing a definition for the public 
programs, and 3 states (with almost no overlap) are developing 
definitions for the proprietary sector. 



Table I I 1-2. Status of Development of a Standard Definition 
for a "Completion" 



NUMBER (PERCENT) OF 
STATES: 


COMMUNITY 
COLLEGES 


PROPRIETARY 
INSTITUTIONS 4 


Having a Definition in Place 


39 (85%) 


26 (90%) 


Developing a Definition 


6 (13%) 


3 (10%) 


Having No Standard 
Definition 


1 ( 2%) 


0 ( 0%) 


Not Collecting Completion 
Information 


6 
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The standard definitions developed by the states differ across 
states, and some states employ more than one definition. The most 
common definition of a completer 1s an Individual who satisfactorily 
completes all program requirements and 1s awarded a degree, diploma, or 
certificate. Thirty-eight states have adopted this definition for the 
public programs, with 16 states having adopted it for the proprietary 
programs. Another definition In use 1s the completion of all phases of 



One state does not have any proprietary institutions. 
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a planned course sequence within a defined program without the award of 
a degree, certificate, or diploma (4 states for public schools, 8 
states for proprietary schools). Individual states have developed 
other definitions, such as the completion of a set proportion of a 
program followed by employment 1n a related job. One state is 
contemplating adopting a definition that Involves a survey of the 
course and program completion intentions of each entering student and 
then computing the actual percent of the goal completed. 

limitations of Coverag e 

It should be noted that of the 39 states that have developed a 
standardized definition of a program completer for students enrolled 1n 
public Institutions, 13 apply the definition and collect data only for 
students awarded an associate degree. Nondegree programs are excluded 
from the data collection efforts of these Institutions. No parallel 
exclusion of students in nondegree programs was found among proprietary 
schools. 

Comparisons across and within states for planning and policy- 
making purposes are difficult, 1f not Impossible, for the following 
reasons: (1) many states do not collect data on the number of students 
completing programs or have no standardized definition of a program 
completer, (2) the definitions and data collection efforts exclude a 
large proportion of vocational education students, and (3) the 
definitions of program completion vary across states. 



12 



Notable Practices 

In reviewing the practices for collecting completion data 
reported by the states, it was noted that the activities of several 
states were distinctive. 

One o* the issues in defining a "completer" has been whether 1t 
is appropriate to assume that an individual who does not complete all 
of the requirements o? a degree, certificate, or other award did not 
accomplish his or her goals. Illinois reported that they were planning 
a pilot effort among public school systems that would determine each 
individual's personal goals, which in turn would be used as the basis 
to determine whether the individual completed them. Community colleges 
In Colorado collect information about applicants' educational 
Intentions (according to one of five categories), with these enrollment 
reasons entered into the state's Student Unit Record Data System. 

In Alaska, rather than imposing a definition from the state 
level, an extensive effort is under way to involve all segments of the 
vocational education community in the development of definitions that 
are useful and practical to collect. 

Numbers of Students Placed 
Another commonly used outcome measure for postsecondary 
vocational education programs 1s the number of students employed after 
they leave the program. Table 1 1 1 -3 shows that the majority of states 
collect some kind of placement data, although the percentages are 
smaller than for those collecting completion data. 
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Table I I I -3 . States Collecting Placement Counts 



NUMBER (PERCENT) OF 
STATES: 


COMMUNITY 
COLLEGES 


PROPRIETARY 
INSTITUTIONS 


Collecting This Information 


33 (64%) 


21 (40%) 


Not Collecting This 
Information 


19 (36%) 


31 (60%) 



Development of a standardized definition of a placement has not 
been finished 1n all states collecting placement counts. Table III -4 
Indicates that a standard definition has been implemented in 79 percent 
of the states collecting placement data from public Institutions, and 
in just over half of the states collecting data from proprietary 
institutions. All of the standardized definitions limit the count of 
placements to students who are classified as program completers. 



Table III-4. States That Have Developed a Standard 
Definition for a Placement 



NUMBER (PERCENT) OF 
STATES: 


COMMUNITY 
COLLEGES 


PROPRIETARY 
INSTITUTIONS 


Having a Definition in Place 


26 (79%) 


11 (52%) 


Developing a Definition 


5 (15%) 


4 (19%) 1 


Having No Standard 
Definition 


2 ( 6%) 


6 (29%) 


Not Collecting Placement 
Data 


19 


31 



The standard definition of a placement developed by a majority of 
states involves an assessment of the relationship of the placement to 
the training received. Only 2 states collecting data from public 
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Institutions and 3 from the proprietary sector do not Impose a 
condition of "training relatedness" before employment after program 
completion can be counted as a placement. Some of these states also 
consider enlisting in the armed services or enrolling in a 4-year 
institution as a placement (12 states collecting data from public 
institutions, 4 states collecting from proprietary institutions). 

Table 1 1 1-5 indicates that the states use a variety of means for 
determining whether a job held by a program completer is related to the 
training received. The most common measure is to allow the former 
students to make the determination as part of a response to a survey. 
Although It is the simplest to implement, this method of determination 
would seem to offer the least valid and reliable information. 



Table I I I -5. Methods for Determining the "Training 
Relatedness" of an Occupation 



NUMBER (PERCENT) OF 
STATES WHERE: 


COMMUNITY 
COLLEGES 


PROPRIETARY 
INSTITUTIONS 


Former Students Make the 
Determination 


23 (70%) 


10 (47%) 


Schools Make the 
Determination 


5 (15%) 


6 (29%) 


Employers Make the 
Determination 


1 ( 3%) 


2 (10%) 


State Agency Makes the 
Determination 


2 ( 6%) 


0 ( 0%) 


A Training Relatedness 
Determination Is Not Made 


2 ( 6%) 


3 (14%) 


1 Placement Information Is 
| Not I Dllected 


19 





IS 
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Notable Practices 

Florida makes extensive use of the Unemployment Insurance 
Employer Wage and Tax Report collected by the state department of 
employment security. By matching the social security numbers of 
postseco.idary vocational program participants against wage data records 
and other automated sources, it is possible to obtain accurate counts 
of the numbers of former program participants employed, and to obtain 
some estimate of their earnings. 

Washington has developed a procedure for assessing whether the 
job a former student holds is related to the training received. They 
have created a matching matrix that compares the courses completed by 
each Individual using the Classification of Instructional Program (CIP) 
codes with the requirements for each occupation, represented by the 
Standard Industrial Classification (SIC) codes. 

Other Performance Measures 

A small number of states collected several outcome measures from 
postsecondary vocational education Institutions in addition to 
completion and placement data. Among these additional outcome measures 
are earnings, gains in basic or job-related skills, and success of 
program graduates in passing licensure examinations. Table I I 1-6 shows 
the relative frequency of the collection of each measure. 

The collection of information on earnings from public 
institutions was reported in 21 states. Seventeen of the states used 
student surveys to obtain this information; 4 states (Florida, 
Illinois Washington, and Utah, with the latter in a pilot test mode) 
used unemployment insurance wage records data. No states reported the 
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routine, comprehensive collection of earnings data for former 
proprietary schools students; California, New Jersey, and Tennessee 
require schools to provide validating Information when claims about 
earnings are used 1n advertising. 



Table III-6. Number of States Collecting Other Outcomes Measures 



OUTCOME MEASURE 


COMMUNITY 
COLLEGES 


PROPRIETARY 

iNSTrrunONS 


Earnings 


21 


3 

(When used in 
advertising) 


Gains in Basic Skills 


13 


0 


Licensure 


12 


2 



A small group of states have elected to Include gains 1n basic 
skills or in the skills specific to an occupation as measures of 
performance for their public vocational institutions. Twelve states 
gather Information on gains in basic skills using standardized tests 
administered prior to enrollment and prior to completion. Six of these 
states also collect information on gains in job skills. One state 
collects only information about gains in job skills. No states 
reported collecting such measures for students attending proprietary 
schools in programs for occupations other than the licensed trades. 

A training program completer's score on the examination for a 
license trade 1s generally used only as a measure of performance for 
the Individual. Only a few states have aggregated the data on the 
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success rates of former students by program or by school to serve as a 
measure of the performance of the education offered. 

Notable Practices 

Colorado, Connecticut, and New Jersey make systematic efforts to 
assess changes In basic skills. A standardized test Is administered 
soon after enrollment and again at the end of the program for students 
1n public Institutions. Connecticut uses the New Jersey Test of Basic 
Skills. Colorado uses the American College Testing (ACT) Program's ACT 
plus program-specific exams. New Jersey administers the New Jersey 
College Basic Skills Placement Tests. 

Computation of Rates 
The performance measures discussed thus far have all consisted of 
counts of the numbers of students In each category. The computation of 
ratios or rates for these measures would greatly enhance the usefulness 
of these measures for comparisons across institutions, for a review of 
a single institution over time, or for the attainment of some 
established standard. However, the use of the outcome measures 
collected to develop rates was not universally observed. Of those 
states that collect completion counts, for example, only 75 percent 
compute a completion rate for the public component's postsecondary 
vocational education and 67 percent for the proprietary component. 
Table 1 1 1-7 shows that placement rates are computed by an even smaller 
proportion of the states: 55 percent for public institutions and 62 
percent for proprietary institutions. 
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Table II I -7. States Computing Completion and Placement Rates 



^ (PERCENT) OF 
STATES: 


COMMUNITY 
COLLEGES 


PROPRIETARY 
WSTITUnONS 


Computing Completion Rate 


34 

(74% of the states 

collecting 
completion data) 


19 

(66% of the states 
collecting completion 
data) 


Computing Placement Rate 


18 

(54% of the states 
collecting placement 
data) 


13 

(62 % of the state? 
collecting placement 
data) 



The computation of a completion or placement rate requires at 
least two elements: a definition of a completion or a placement, and 
agreement on the base to which the number of completers or placements 
is to be compared. Establishing a consensus for a definition of the 
base number for these rates and obtaining the appropriate data appear 
to be areas 1n which state", that are seeking to measure program 
outcomes are experiencing difficulty. 

For completion rates, most states use the number of students 
entering a program at a given time (e.g., 2 years earlier for a program 
that typically takes 2 years to complete). Among the variations used 
to construct the base are to exclude those still 1n school, to exclude 
an estimated proportion of entering students who will leave the program 
and later return to finish 1t, or to use only the number of students 
completing a predetermined number of courses as the threshold for 
Inclusion in the computation. One state, for example, only Includes 
students who have completed 10 vocational credits in any measures of 
completion or placement for Its public institutions. 
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The primary difficulty In developing a formula for computing a 
placement rate was establishing a consensus on which categories of 
unemployed program completers to exc"; J e. The schools argue that It is 
appropriate to exclude from the calculations students who have become 
111, are pregnant, or are single parents of young children. The 
schools also Indicate that students who have decided not to seek 
employment should not be Included, and that students who do not use the 
school's placement service—which 1s argued to be an Integral part of 
the career preparation component of their programs --should not be 
counted In measures of the performance of these programs. 




IV. DATA COLLECTION AND VERIFICATION 

Outcome measures, almost by definition, focus on events 1n the 
careers of program completers after they leave the school. With the 
exception of program completion counts and rates, the data on program 
outcomes must be obtained from sources beyond the records schools 
maintain for their own administration. The most frequently used source 
is surveys of former students, followed by surveys of employers. The 
efforts to collect Information through these sources raise serious 
questions about the survey design, representativeness of the former 
students who respond, the validity and reliability of the Information 
collected, and the burden this effort places on the respondents and the 
schools. Some states use third-party data, such as state Unemployment 
Insurance Wage Records data, as a way to obtain reliable Information at 
a low level of effort, and a significant number of other states 
reported an Interest in using such sources. 

Data Collection 

Surveys 

Table IV- 1 shows that 21 states require public postsecondary 
vocational schools to conduct graduate surveys to collect a variety of 
information. In Maryland and Washington, the state oversight agency 
surveys graduates of these schools directly. Florida currently 
requires proprietary schools to conduct a survey of students. Five 
additional states soon will implement surveys of graduates for public 
schools, and 3 plan to Implement surveys of proprietary schools. 
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Table IV-1. Data Sources for information Other Than Completion Data 



DATA SOURCE 
(Some States Use Multiple 
''- Sources) -\ \ 


COMMUNITY 
COLLEGES 


PROPRIETARY 
INSTITUTIONS 


Schools Required to Survey 
Completers 

j 

II 


21 

(64% of states 
collecting 
information beyond 
completion) 


1 

(5% of states 
collecting 
information beyond 
completion) 


State Agency Surveys 
Comoleters 


2 

(6%) 


0 


Schools Required to Survey 
Employers 


11 

(33%) 


2 

(10%) 


State Wage-Data or IRS 
Data (Excludes Pilot Studies 
or Demonstration Programs) 


5 

(15%) 


0 



A major component of the surveys 1s the current employment status 
of the respondent. Former students who are not employed are asked 
questions to determine whether they are pursuing employment. Those who 
are employed are asked jto complete questions on the relationship of the 
current job to the training received. In 17 states, the employed 
graduates are asked for current or Initial wage Information. Other 
questions frequently Included are measures of job satisfaction and 
ratings of the quality of the training program. 

Eleven states require public Institutions to conduct surveys of 
the employers of their graduates. However, only Florida and Oregon 
require proprietary institutions to conduct employer surveys. In 
nearly all of these states, the employers are asked to rate whether the 
training received by the graduates is related to the current job. The 



employers also frequently are asked to rate the job performance of the 
graduate and to rate the quality of the training program. 

State Wage and IRS Data 

During the assembly of Information for this study, Interest was 
expressed in the use of third-party data for some or all performance 
measures. The most frequently mentioned sources were the state 
Unemployment Insurance Wage Records and insurance files. Currently, 5 
states (Florida, Illinois, Kansas, Kentucky, and Washington) use these 
data on a regular basis as a source or performance measures for public 
institutions. Four additional states are said to be near 
implementation of the regular use of these data for performance 
measures. Four states are using either state wage data, IRS data, or 
both in pilot studies of the development of performance measures or the 
establishment of student data. Outcome measures that can be gathered 
for individual graduates of training programs from these data are 
Information on placement status and some Information on earnings. 

The states make other uses of these data sources that are not 
related to performance measures. At least one state uses the combined 
wage and unemployment data to develop labor force projections. A 
number of states use the information as part of loan default management 
programs to track graduates. 

Data Verification 
The majority of the states that collect performance data report 
little activity to verify the validity of the data collected. A 
summary of the reports of verification efforts is presented 1r 
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Table IV-2. The verification activity most often described was the 
review of records during periodic on-site reviews of program operations 
and records. Most of these reviews are annual, but in some cases as 
many as 8 years pass between reviews. These reviews are conducted for 
public institutions in 17 states, and for proprietary institutions in 
11 states. 



Table IV-2. Data Verification Methods 



VERIFICATION 
METHOD 


COMMUNITY 
COLLEGES 


PROPRIETARY 
INSTITUTIONS 


Review of School Records 


17 

(37% of states 
collecting any 

outcome 
information) 


11 

(38% of states 
collecting any 

outcome 
information) 


Direct State Contact with a 
Sample of School Survey 
Respondents 


0 


4 

(14% of states 
collecting any 

outcome 
information) 


Verification of Advertising 
Claims 


0 


11 

(38% of states 
collecting any 

outcome 
information) 



The purpose of the verification effort during the on-site reviews 
is to certify that the data submitted to the state matches the data 
contained In the school's files. With the exception of data on program 
completion, this type of review cannot be said to establish the 
validity of that information. It would not reveal whether biases were 
Introduced through the manner 1n which the survey was conducted, the 
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manner 1n which responses were handled or edited, or a variety of other 
sources. 

Several efforts go beyond this type of review to evaluate the 
validity of the performance data: 

• Four states (Florida, Minnesota, Oregon, and Rhode Island) seek 
to assess the validity of the data reported by proprietary 
schools by making direct contact with a sample of the graduate or 
employer respondents to the surveys conducted by the proprietary 
schools. Only South Carolina assesses this validity for public 
schools. 

• Eight states, two of which are 1n the pilot test phase, reduce 
the likelihood that the data are biased by the actions of the 
Institutions by conducting surveys of graduates or employers 
directly, or by using data from Independent third parties. 

• Eleven states conduct efforts to verify data from the proprietary 
schools when those schools make claims about performance measures 
1n their advertising. 



Analyses 

The analyses conducted by the states with the Information 
collected vary in complexity. Table IV-3 shows the relative frequency 
with which different kinds of analyses are conducted. Overall, the 
majority of the states do not conduct analyses that enhance the ability 
of the policymaker or consumer to differentiate between the outcomes of 
alternative programs or Institutions. In most Instances, the analyses 
that were reported focus on placement rates. The primary analysis of 
completion rates that was reported was a quick examination to see that 
there was no significant decrease from prior years. 
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Table IV-3. Analyses of the Data Collected 
(Multiple Responses Counted) 



NUMBER (PERCENT) OF 

States conducting 
each type of analysis 


COMMUNITY 

' COLLEGER 


PROPRIETARY 

msrrmmoNS 


Comparison with an 
Established State Standard 


17 (37%) 


8 (28%) 


Comparison of Placement Data 
with State Employment Trends 


5 (11%) 


3 (10%) 


Ranking of Schools on Key 
Indicators 


3 ( 7%) 


0 ( 0%) 


Review of Each Program 


10 (22%) 


r f o%) 

L 


Evaluative Narrative 


5 (11%) 


0 ( 0%) 


Employment Profile by 
Geographic Region 


6 (13%) 


0 ( 0%) 


Listing of Data Reported by 
Schools 


8 (17%) 


4 (14%) 


States Collecting Any Outcome 
Data 


46 


29 



The analysis that was reported most often to have been conducted 
using the data collected 1s a comparison of a critical outcome 
Indicator to a standard. A critical outcome Indicator analysis 1s done 
for public Institutions in 17 states and for proprietary schools in 8 
states. Several examples are provided below: 

• California: At least a 60 percent completion rate and at least 
a 70 percent placement rate 6 mot'Vis after completion must be 
maintained for each proprietary school program, or the program 
will fse discontinued (planned for 1991). 

« 

• Florida: An Improvement of 20 percent each year 1s required in 
the number of students completing each public school program. 
Also, if the placement rate for a program in a public institution 
falls below 70 percent for 3 consecutive years, the program loses 
state funding. 




• Illinois: If the completion rate for a proprietary school falls 
below 50 percent of the industry average in the state, the 
institution is placed on probation for 1 year. If the rate is 
not above this average the fol'owing year, the school's license 
can be revoked. 

• Kansas: Public schools are required to have a 70 percent 
placement rate for students within 2 years after graduating. 
This review leads to an agreement on prograir. improvements and 
assistance from the state. Repeated failure to meet the critical 
standard could lead to discontinuation of a program. 

• Oregon: If a proprietary school fails to maintain completion and 
placement rates of at least 75 percent of the state average, it 
will be put on probation for up to 1 year. If unable to maintain 
at least this 75 percent of state average for 2 consecutive 
years, the state may revoke the school's license. 

The second most common analysis done is simply to format the data 

as it is reported without comparison to a critical Indicator, 

comparison to employment trends, or even comparison to data submitted 

by the school in previous years. This approach 1s taken in 8 states 

for the data submitted by public schools and 4 states for proprietary 

schools. 

Five states compare placement data submitted by public schools to 
statewide employment trends in each occupation. Three states perform 
the same analysis for proprietary schools. 

Three states produce rankings of all of the programs in public 
schools by key indicators. 

Ten states conduct a review tor each program offered by each 
public postsecondary vocational school, attempting to Identify 
strengths and weaknesses. Five states prepare evaluative narratives 
for each school. These reviews are conducted in an advisory capacity, 
with the state agency and the institutions in a partnership to maximize 
the quality of the education offered. 



27 

39 



Six states aggregate the data submitted by Individual schools to 
develop profiles either geographic region within the state or by 
trade. The analyses seek to Identify areas or trades that present 
employment opportunities and those 1n which employment 1s problematic. 
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V. USES OF THE OUTCOME DATA 

Obtaining clear, exhaustive Information on the uses made of the 
performance data collected was one of the hardest elements to gather 
for the overview of state activities. It appears that those within the 
oversight agencies who are Involved 1n the collection of the data do 
not have full knowledge of how the Information Is used, particularly 
when it 1s forwarded to other agencies. 5 In these situations, the 
staff responsible for the data collection were often unsure of the uses 
of these data by the other agencies. In general, the uses of the data 
reported fell Into three categories: consumer rights, licensure, and 
a variety of activities that are loosely termed as management. The 
relative frequency of each use 1s shown 1n Table V-l. 

For this overview, a state was said to use the data for consumer 
rights when the data were made available to the general public, either 
through direct mailings or when available statewide 1n public 
libraries, and when the Information available contained some evaluative 
Information. The evaluative Information could be explicitly provided 
by the oversight agency, or they could provide the background 
information necessary for such a judgment --such as a statewide average 
or the statement of performance standards. Some of the sources 
contacted considered consumer rights uses to include the use of data by 
the oversight agency to close down poor performing programs, to 
establish new programs in growing occupations, or to Improve the 



5 For example, the staff member of a postsecondary education commission who 
assemble data from vocational schools might deliver his or her findings to the 
state education department and to the state department of labor. 
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performance of programs to benefit consumers. This classification was 
not adopted for this report because the consumer was not directly 
Involved 1n the decision on what action to take. 



Table V-l. Number (Percent) of States Reporting Various 
Uses of the Performance Measures Collected 



USBS OF PERFORMANCE 
'^^C MEASURES 


community 

COLLEGES 


PROPRIETARY 
INSTITUTIONS 


Consumer Rights 


14 (30%) 


8 (28%) 


Licensure 


0 ( 0%) 


17 (59%) 


State Funding Decisions 


22 (48%) 


0 ( 0%) 


Expansion/Contraction of 
Programs by Occupation 


21 (46%) 


0 ( 0%) 


Program Review to 
Improve Operations 


13 (28%) 


8 (28%) 


Determination of Need for 
Corrective Actions 


18 (39%) 


9 (31%) 


Performance-Based Allocation 
of Supplemental Funds 


1 ( 2%) 


0 ( 0%) 


Number of States Collecting 
Any Outcome Information 


46 


29 1 



Fourteen states make the information collected from programs in 
public institutions available for consumer rights uses. Four of these 
states also make the Information from proprietary schools available for 
consumer rights issues. Four additional states make only the 
proprietary school Information available to consumers. 

One of the responsibilities of the state agency assigned 
oversight responsibility for proprietary schools is the periodic 
consideration of whether to grant each school r license to continue to 
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operate in the state. Seventeen states reported that outcome 
measurement data were part of the materials examined when reviewing 
applications for license renewals. Schools falling below established 
standards were subject to more Intensive reviews, being placed on 
probation, or being denied a license. 

Management uses Involve a number of dimensions. One use 1s 
decisions made by the governor and state legislature of funding levels 
for' public Institutions, Including the decision to "defund H an 
Institution or an occupational program. This was reported to be a 
major use in 22 states. Another dimension is the decision about 
whether to expand or contract opportunities to enroll in a training 
program in a particular occupational area. This could be done by 
changing the size of existing Institutions, or by creating new 
Institutions or closing existing ones. This is a use of the data from 
public institutions in 21 states. 

Program review is another dimension of management uses of the 
data. Two basic kinds of program review are conducted. One 1s to aid 
Institutions in making sound management decisions about program 
offerings, enrollment size, and the quality of training presented. 
These are generally conducted by the oversight agency as an aid to and 
1n partnership with the institutions. This was a focus in 13 states 
for public institutions and 1n 8 states for proprietary schools. 
Another category of program review is to assess the quality of the 
school with the aim of imposing corrective actions to improve program 
offerings, or to identify schools experiencing problems that call for 
more intensive contact and review by the state agency. This was a 
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focus 1n 18 states for public schools and 1n 9 states for proprietary 
schools. 

Notable Practices 

Tennessee makes use of the outcome measures 1t collects to make 
performance- based funding decisions when distributing supplemental 
funds to public Institutions over and above their base allotments. 

Reports Produced 

The most common reporting format listed by the states 1s to 
prepare a descriptive Institutional profile that contains the outcome 
measures for each program within each school but that does not contain 
evaluative comments, ranking, or comparison to a standard. This 1s at 
least one of the reports generated for public Institutions in 30 states 
and for proprietary institutions in 12 states. 

Another reporting format is to present Information by program and 
school but which includes some evaluative measure of performance, 
whether it 1s a summary of problems encountered, comparisons to the 
state average for this type of training program, a ranking, comparison 
to a previous year's data, or comparison to a standard. This type of 
report is prepared for public institutions in 20 states and for 
proprietary Institutions in 7 states. 

Other kinds of reports Include those that provide directories of 
schools, give the programs offered, and indicate licensing status 
(often excluding proprietary schools that are not licensed). These are 
prepared for public schools in 3 states and for proprietary schools In 
8 states. 
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In 5 states, data from the public schools are used to prepare 
profiles by trade or geographic area, giving Indications of where 
employment opportunities are to be found. Data from proprietary 
schools are included in these reports in 3 of these states. 

Another major category of reports is responding to ad hoc 
requests from the governors, legislatures, or other government 
agencies. This is the major reporting activity for data from public 
Institutions in 7 states. In addition, 6 states report that the data 
are placed on-Hne for inquiries about Individual schools or for 
aggregate analyses. 

Nine states indicate that although they collect performance data 
from the public institutions, no formal reports are produced from the 
data. Ten states do not prepare a report from data collected from 
proprietary schools. 

Report Recipients 
Table V-2 presents a taly of the recipients of reports generated 
by the oversight agencies. It should be kept in mind that most of the 
agencies prepare more than one kind of report, and the reports do not 
have the same pattern of dissemination. Overall, the relative 
frequency of report recipients were the same for both the data 
collected from community colleges and from proprietary schools. 
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Table V-2. Recipients of Reports Based Upon 
Performance Data 



REPORT RECIPIENT 


NUMBER (PERCENT) STATES SENDING 
REPORTS BASED UPON PATAiP^JCMt' 


COMMUNITY 
COLLEGES 


PROPRIETARY 


Oversight Acencv for 
Internal Use Only 


7 (15%) 


4 (14%) 


Schools Submitting 
Data 


28 (61%) 


11 (38%) 


Other State Agencies, 
Boards, Commissions, 
Etc. 


17 (37%) 


3 (10%) 


Governor and State 
Legislature 


27 (59%) 


9 (31%) 


Secondary Sdhools 


9 (20%) 


2 ( 7%) 


General Public or 
Public Libraries 


16 (35%) 


8 (28%) 


Number of States 
Collecting Any 
Outcome Data 


46 


" 1 



All of the agencies that collect outcome information use those 
data internally. The most frequently reported external recipients of 
the data were the schools that had submitted the information, the 
governors and state legislature, and other agencies of the state 
government. Approximately 15 percent of the states reported that the 
data collected were used only by the collecting agency. 



9 

ERIC 



34 



46 



VI. TRENDS IN THE DEVELOPMENT OF STATE POLICY 

Two of the striking features of the information gathered for the 
overview were how recently changes had been enacted in many states and 
how frequently descriptions of new procedures (that had been enacted 
but were to be implemented at a later date) were encountered. In fact, 
some of the difficulties frequently encountered in assembling the state 
overview were that many of the information sources were focused on 
descriptions of changes not yet implemented, or proposed changes not 
yet approved, and were reluctant to provide materials or answer 
questions about present practices. An information source in one state 
called back three times in 2 months to report the passage of new 
legislation related to outcome measurement. 

Table VI- 1 summarizes the pace of change reported during the 
assembly of the overviews. Few states were found to have Initiated 
policies or made major changes to existing policies for the collection 
of performance data more than 2 years previously. (Several mentioned 
the termination of the Vocational Education Data System [VEDS] 1n the 
mid-eighties.) Overall, 19 states either initiated the collection of 
outcome measures from postsecondary vocational Institutions or made 
major changes to increase the amount of Information collected from 
these schools within the last 2 years. Four of those states initiated 
policies or made changes in the collection of outcome measures from 
both public and proprietary institutions. Eleven states addressed only 
public institutions, and an additional 4 states addressed the 
proprietary schools only. 
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Table VI- 1. The Pace of Change in State Pjlicies 



TIME PERIOD OF 
CHANGE 


NUMBER (PERCENT) OF STATES MAKING 
CHANGES 


COMMUNITY 
COLLEGES 


PROPRIETARY 
INSTITUTIONS 


Change Implemented 
Within the Last 2 
Years 


15 (29%) 


7 (13%) 


Legislation for Change 
Passed, but 
Implementation to 
Occur Later than 
Spring 1990 


7 (13%) 


2 ( 4%) 


Change Under Serious 
Consideration fBills 
Introduced, 
Regulations Drafted) 


8 (15%) 


6 (12%) 


Discussions of Changes 
Under Way 


6 (12%) 


6 (12%) 


No Apparent Activity 


16 (31%) 


31 (60%) 1 



Legislation has been passed or regulations have been approved 
that call for changes to be implemented for public institutions in 7 
states and for proprietary institutions in 2 states, all at some date 
later than spring o f 1990. Activities that are being carried out 
during the waiting period include developing definitions of performance 
indicators, determining the format for data collection, determining 
what analyses and reports will be produced, and making arrangements for 
access to third-party data, such as state Unemployment Insurance Wage 
Records data. 



9 

ERIC 



36 

48 



In another group of states, bills have been Introduced in the 
legislature or regulations drafted that are reported to have good 
chances of passage and Implementation. Public Institutions are the 
focus of these efforts in 8 states, and proprietary institutions are 
the focus in 6 states. In a final group of states, serious 
consideration and discussion of the Implementation of efforts to 
collect outcome measures were reported to be under way. Six states are 
discussing such measures for public institutions, and 6 states are 
considering them for proprietary schools. 

In 15 states, there was no apparent activity that would result in 
a change in the collection of performance measures for public 
postsecondary vocational schools, and in 31 states there were no 
reports of changes for proprietary schools. 
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VII. SUMMARY AND CONCLUSIONS 

This report has presents an overview of the activities conducted 
by the states to collect and analyze outcome performance data from 
postsecondary vocational education institutions nationwide. It was 
found that most states have undertaken limited activities to assess 
performance outcomes, but few states have employed multiple measures of 
performance, and most have had difficulty analyzing the data that they 
collect or in using it for program improvement or policy. It also was 
found that many states are making efforts to increase the quality and 
comprehensiveness of the Information collected and to make greater use 
of the Information collected. 

The overview also shows that there is wide variation in the level 
of effort individual states are making to assess program outcomes and 
in the sophistication of the approach that has been implemented. There 
is wide variation among the states in every aspect of the outcome 
measurement process, Including the decision on what to measure, how to 
collect the necessary information, how to analyze it, what use to make 
of the data, and with whom to share the results. 

The practices of the states are changing rapidly. Few of the 
states that collect outcome measures were found to have had an outcome 
measurement policy in place that was unchanged for more than 2 years. 
With regard to public postsecondary vocational institutions, 15 states 
either had implemented new policies for collecting outcome data or had 
made changes in their policies to either expand the kinds of 
information collected or improve the methods of collection. Seven 
states have passed legislation calling for similar changes, but they 
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will be Implemented at some point after the spring of 1990, the time 
the Information for this overview was collected. In other states, 
bills have been Introduced or serious consideration 1s being given to 
the adoption of such changes. In only 16 states was no apparent 
activity found in this area. 

Fewer efforts to collect outcome measures are under way for 
proprietary schools. No efforts to Implement or improve the collection 
of outcome measures were found 1n 31 states. Seven states have 
Implemented policies or made changes to improve the collection of 
outcome measures during the last 2 years, and two additional states 
have passed legislation that will be implemented after the spring of 
1990. 

The most frequently collected outcome measure was the number of 
students who completed a training program (46 states for public 
postsecondary schools, 29 for proprietary schools). The next most 
frequently completed measure collected at the state level was whether 
program completers were placed in Jobs (33 states for public schools, 
21 for proprietary schools). In nearly every instance, efforts were 
made to determine whether the placement was related to the training 
received, usually by asking the program completer to make that 
judgment. Such outcome measures as earnings and gains in basic skills 
or job knowledge were collected in smaller numbers of states. For each 
outcome measure, policies were significantly more likely to be in place 
to collect these data for public schools than for proprietary schools. 

Efforts to establish the validity of the data collected were 
found in few states. Analyses of the data tended to focus on 
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comparisons of key Indicators with established standards, such as a 
minimum percentage of completers to be placed, or on the construction 
of a profile by occupational program with determinations of performance 
that was unusually high or poor. 

The uses for which the outcome measures were collected also 
varied widely across states. The most frequent use was to conduct a 
program review to either provide the institution with the information 
necessary to decide on programs to offer and means to Improve quality, 
or to provide the oversight agency with an Indicator that a school was 
performing poorly. A number of states also made the information 
available to the public with an explicit goal of providing information 
necessary for informed consumer decisions. Fourteen states made 
outcome measures for public schools available for use by consumers, and 
8 states made measures available from proprietary schools. 
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